{ "cells": [ { "cell_type": "markdown", "id": "0e7cb5e6", "metadata": {}, "source": [ "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " Try in Google Colab\n", " \n", " \n", " \n", " \n", " Share via nbviewer\n", " \n", " \n", " \n", " \n", " View on GitHub\n", " \n", " \n", " \n", " \n", " Download notebook\n", " \n", "
\n" ] }, { "cell_type": "markdown", "id": "ee615feb", "metadata": {}, "source": [ "# **FiftyOne 3D Detections**" ] }, { "cell_type": "markdown", "id": "afae3fe7", "metadata": {}, "source": [ "## Create Your First 3D Detection Label" ] }, { "cell_type": "markdown", "id": "61fff01b", "metadata": {}, "source": [ "We will use [quickstart-groups](https://docs.voxel51.com/user_guide/dataset_zoo/datasets.html?highlight=quickstart%20groups#dataset-zoo-quickstart-groups) to gather some LIDAR point cloud data to create our labels:" ] }, { "cell_type": "code", "execution_count": 18, "id": "9640f3e0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset already downloaded\n", "Loading 'quickstart-groups'\n", " 100% |█████████████████| 600/600 [1.7s elapsed, 0s remaining, 365.8 samples/s] \n", "Dataset 'quickstart-groups' created\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "Could not connect session, trying again in 10 seconds\r\n", "\n" ] } ], "source": [ "import fiftyone as fo\n", "import fiftyone.zoo as foz\n", "from fiftyone import ViewField as F\n", "\n", "dataset = foz.load_zoo_dataset(\"quickstart-groups\") \n", "dataset.group_slice = \"pcd\"\n", "session = fo.launch_app(dataset)" ] }, { "cell_type": "markdown", "id": "509bd1c1", "metadata": {}, "source": [ "Ah! Those pesky error messages are no good. Since it is point cloud data, we have no thumbnail to display. We can change this by utilizing orthographic projection. This will create a birds eye view of our dataset and add the projection to a slice of our group as well. To learn more, look [here](https://docs.voxel51.com/api/fiftyone.utils.utils3d.html?highlight=ortho#fiftyone.utils.utils3d.compute_orthographic_projection_image)!" ] }, { "cell_type": "code", "execution_count": 19, "id": "00103186", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 100% |█████████████████| 200/200 [11.9s elapsed, 0s remaining, 16.9 samples/s] \n", " 100% |█████████████████| 200/200 [76.2ms elapsed, 0s remaining, 2.6K samples/s] \n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import fiftyone.utils.utils3d as fou3d\n", "\n", "size = (-1, 512)\n", "fou3d.compute_orthographic_projection_images(\n", " dataset,\n", " size,\n", " \"/tmp/proj\",\n", " shading_mode=\"height\",\n", " out_group_slice=\"proj\"\n", ")\n", "\n", "session.view = dataset.view()\n" ] }, { "cell_type": "markdown", "id": "630795b5", "metadata": {}, "source": [ "Much better!" ] }, { "cell_type": "markdown", "id": "714ed522", "metadata": {}, "source": [ "## Detections" ] }, { "cell_type": "markdown", "id": "ed687cfa", "metadata": {}, "source": [ "To add a [3D detection](https://docs.voxel51.com/user_guide/using_datasets.html#d-detections), we create a `fo.Detection()` label and add the label, location in x,y,z, rotation in radians (-pi, pi), and dimensions in width, length, height" ] }, { "cell_type": "code", "execution_count": 21, "id": "3a81b1f5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "dataset.group_slice = \"pcd\"\n", "sample = dataset.first()\n", "\n", "bounding_box = fo.Detection(\n", " label=\"example\",\n", " location=[0,0,0],\n", " rotation=[0, 0, 0],\n", " dimensions=[3,3,3]\n", " )\n", "\n", "sample[\"example\"]= bounding_box\n", "sample.save()\n", "dataset.save()\n", "view = dataset.filter_labels(\n", " \"example\", F(\"label\").is_in([\"example\"])\n", ")\n", "\n", "session.view = view" ] }, { "cell_type": "markdown", "id": "1ba48c07", "metadata": {}, "source": [ "Play around with changing values to get a feel for the coordinate system:" ] }, { "cell_type": "code", "execution_count": 22, "id": "f8ed01a3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sample = dataset.first()\n", "\n", "bounding_box = fo.Detection(\n", " label=\"example\",\n", " location=[0,2,4],\n", " rotation=[0, 0, 0],\n", " dimensions=[1,2,3]\n", " )\n", "\n", "sample[\"example\"]= bounding_box\n", "sample.save()\n", "dataset.save()\n", "\n", "view = dataset.filter_labels(\n", " \"example\", F(\"label\").is_in([\"example\"])\n", ")\n", "\n", "session.view = view" ] }, { "cell_type": "markdown", "id": "949cf86e", "metadata": {}, "source": [ "Lets try experimenting with rotation now! Tweak the values to try to get the box to point in different directions!" ] }, { "cell_type": "code", "execution_count": 23, "id": "8c09c316", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "from fiftyone import ViewField as F\n", "# x = left+right = orange | y = forward/back = green | z = up+down = blue\n", "bounding_box = fo.Detection(\n", " label=\"example\",\n", " location=[0,0,0],\n", " rotation=[np.pi/4, np.pi/4, 0],\n", " dimensions=[3,2,1]\n", " )\n", "\n", "sample[\"example\"] = bounding_box\n", "sample.save()\n", "dataset.save()\n", "view = dataset.filter_labels(\n", " \"example\", F(\"label\").is_in([\"example\"])\n", ")\n", "\n", "session.view = view" ] }, { "cell_type": "markdown", "id": "434cbbe2", "metadata": {}, "source": [ "Another type of 3D label is [polylines](https://docs.voxel51.com/user_guide/using_datasets.html#d-polylines)! They are created by defining the coordinates in a list for each point on the line. Here we make a triangle:" ] }, { "cell_type": "code", "execution_count": 24, "id": "5b1b90c8", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "label = \"lane\"\n", "\n", "# A list of lists of `[x, y, z]` points in scene coordinates describing\n", "# the vertices of each shape in the polyline\n", "points3d = [[[1, 1, 1], [0, 2, 2]], [[0, 2, 2], [-1, 1, 1]], [[-1, 1, 1], [1, 1, 1]]]\n", "\n", "# A set of semantically related 3D polylines\n", "polyline = fo.Polyline(label=label, points3d=points3d,)\n", "\n", "sample[\"polylines\"] = polyline\n", "sample.save()\n", "dataset.save()\n", "view = dataset.filter_labels(\n", " \"polylines\", F(\"label\").is_in([\"lane\"])\n", ")\n", "\n", "session.view = view\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 5 }